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This application is submitted in the names of inventors Geoffrey Foote and David 
Gillespie, assignors to Synaptics, Incorporated, a California Corporation. 

METHOD AND APPARATUS FOR CONTROLLING A DISPLAY OF DATA 

ON A DISPLAY SCREEN 

SPECIFICATION 
BACKGROUND OF THE INVENTION 

The invention relates generally to smart compact devices, and specifically 
to the display and manipulation of information on such devices by the use of a pen 
input system capable of sensing the proximity of the pen. 

There are now many computerized devices capable of storing and accessing 
large amounts of data that employ quite compact displays. Examples of such smart 
compact devices are smart handheld devices, such as PDAs and smartphones. 
Other smart compact devices may be kiosks or other equipment that is physically 
large, but with displays that are compact compared to those of traditional 
computers. These smart compact devices are commonly used for storing and 
accessing personal data such as contact addresses and calendar information. A 
smart compact device has an operating system, running on a microprocessor, with 
a display screen and an activating object, such as a finger, pen, and the like, used 
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with a pointing device. The pointing device is used to control the actions of the 
operating system through a user interface. The display screen is typically an LCD 
module, and the pointing device is, for example, a resistive touchscreen. The 
operating system will typically be an event driven operating system with a 
5 graphical user interface, such as a windowing operating system, but may also be a 
text-based user interface. 

A graphical user interface typically is made up from screen objects, which 
can be control objects, such as scroll bars, combo boxes, buttons and drop down 
lists; data objects, such as text windows and bitmapped images; and combinations 

10 of the two, such as hyperlinks within text. The user can store and retrieve 
information from the smart compact device by manipulating the screen objects of 
the graphical user interface with the pointing device. Control objects often have a 
control action associated with them, such as closing a window, or opening a file. 

The graphical user interface on a smart compact device is often derived 

15 from graphical user interfaces that are well known on desktop computers, where 
the pointing device is often a mouse or a touchpad. A key difference between 
desktop computers and smart compact devices is that the display screen on a smart 
compact device is typically 320x240 pixels or less in size. This small size makes it 
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difficult to present the data stored in the smart compact device in an easy to use 
way. 

A second key difference is that the graphical user interface of a desktop 
computer is typically driven by a pointing device that is capable of generating both 
5 X and Y position data, and one or more bits of status information. An example of 
this status information is mouse button state. The pointing device on a smart 
compact device can usually only provide X and Y position data in one state, for 
example the position where the user has touched a resistive touchscreen. 

Techniques that are used in the graphical user interfaces of desktop 
10 computers to manage large amounts of data on a display screen include multiple 
overlapping windows and pop-up data, which provides extra information about 
control and data objects in a graphical user interface when the mouse pointer is 
positioned over them. An example of an implementation of pop-up data is 
Microsoft's ToolTips. 

15 Multiple overlapping windows are of little use on a smart compact device, 

since the overhead of window borders, title bars etc., become prohibitive on a 
small display screen. Pop-up data relies on the pointing device being able to 
provide at least two states with X and Y position data, so that a screen object may 
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be pointed to trigger the pop-up data, and then selected. Existing pointing devices 
on smart compact devices do not have this capability, and so pop-up data is 
difficult to implement in a user-friendly manner. 

Attempts have been made to implement pop-up data on smart compact 
5 devices using a pointing device that has only one X and Y position state. 
Microsoft's PocketPC generates pop-up data in some situations if the pointing 
device is held on a control object for a period of time. An alternative 
implementation is described in Patent EP 0 996 052 A2, where tapping the 
pointing device switches modes to display pop-up data. Neither of these 

10 approaches is intuitive for the user, and so pop-up data implemented in this way 
detracts from the usability of the smart compact device. 

A further challenge with the user interface of a smart compact device is that 
smart compact devices, unlike desktop computers, are generally used in a variety 
of environments, often ease of use and quick access to data is of paramount 

15 importance. A standard measure of the usability of the user interface of a smart 
compact device is the number of taps or other discrete actions that a user must 
make to perform a specific task e.g. dialing a phone number. 
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Pointing devices that are capable of generating X and Y position data while 
reporting several status bits are well known in desktop computers. In addition to 
the standard mouse, inductive graphics tablets are common. Also, inductive pen 
sensing systems have been used for pen-based tablet computers, which are 

5 characterized by a large form factor. The use of an inductive pen system in a small 
form factor device, such as a PDA, has also been disclosed in WO 00/33244. In 
all these cases the pen system is capable of generating X and Y position data with 
several status bits, where the status bits may depend on whether the pen is touched 
to the screen or not. The ability to report X and Y position while not touching the 

10 screen is referred to as proximity sensing. 

In these prior art systems, the proximity information has only been used to 
mimic the status information available from mouse buttons. No inventive 
extensions to the User Interface of either desktop computers or smart compact 
devices have been made that use proximity information in a novel way to enhance 

15 usability. Specialized niche applications, such as varying spray density of the 
aerosol tool in a graphics package, are the extent of the applications to date. 
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BRIEF DESCRIPTION OF THE INVENTION 
The present invention provides a means for manipulating the presentation 
of data in the user interface of a smart compact device, using information provided 
by the smart compact device's pointing device. 

The invention allows a screen object to be displayed on the display screen, 
and pop-up data associated with the screen object to be accessed. The user 
accesses the pop-up data by manipulating the pointing device in an intuitive 
manner. The novel use of pop-up data on a smart compact device to allow simple 
and easy access to large amounts of data is a key advantage of the invention. 

It is a further advantage that the invention is intuitive in use for both novice 
and experienced device users. 

BRIEF DESCRIPTION OF THE DRAWING FIGURES 
Referring now to the figures, wherein like elements are numbered alike: 
FIG. 1 is a diagram showing a typical window on the display screen of a 

computer, with a typical screen object. 

FIG. 2 is a diagram showing pop-up data, which is displayed when the 

pointing device has been held over the screen object for a short period of time. 
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FIG. 3 is a diagram illustrating the present invention. 

FIG. 4 shows a typical screen object in the form of a hyperlink. 

FIG. 5 shows the same screen object, and its associated pop-up data 20, that 
has been triggered by holding the activating object above the hyperlink. 
5 FIG. 6 shows a typical small display screen on a smartphone showing 

device status information and a window with directory information containing 
screen objects such as the contact name, the contact company and an icon intended 
to trigger the dialing process. 

FIG. 7 shows an alternate small display screen on a smartphone, where the 
10 contacts do not have a dial icon associated with them. 

FIG. 8 shows typical data stored for each contact entry in a smartphone 
including both detailed contact information, and control objects in the form of 
icons which allow dialing the contact, sending email, or returning the display to 
the directory listing of FIG. 7. 
15 FIG. 9 illustrates a prior art process of dialing a contact from the directory. 

FIG. 10 illustrates a typical prior art process for dialing a number when the 
contact name may be ambiguous. 
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FIG. 1 1 shows a process for dialing a number according to the present 
invention. 

FIG. 12 shows a process for dialing a number according to the present 
invention in the case of ambiguous names. 

FIG. 13 illustrates a display according to an alternative embodiment, in 
which the pop-up data does not have to take up the whole display screen of the 
smart compact device, but can be displayed on a part of the display. 

DETAILED DESCRIPTION OF THE INVENTION 
Persons of ordinary skill in the art will realize that the following description 
of the present invention is illustrative only and not in any way limiting. Other 
embodiments of the invention will readily suggest themselves to such skilled 
persons having the benefit of this disclosure. 

The prior art of pop-up data on a desktop computer is shown by FIGS. 1 
and 2. FIG. 1 shows a typical window on the display screen 12 of the computer, 
with a typical screen object 10. FIG. 2 shows the pop-up data 14, which is 
displayed when the cursor associated with a pointing device (a mouse) has been 
held over the screen object 10 for a short period of time. 
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The invention is shown diagrammatically in FIG. 3. In an illustrative 
embodiment, the pointing device may be an inductive pen input system, 
comprising a pen sensor 18 and a pen control system 20, such as that described in 
publication WO 00/33244, entitled "Position Sensor". This system has the key 

5 advantage that it can provide X and Y position data both when the activating 
object is touching the display screen and when it is held in close proximity to, but 
not touching, the display screen, and is able to distinguish between the two states. 
Alternatively, the first state may require a firm touch while the second "non- 
touching" state may occur even when the activating object is lightly touching the 

10 display screen. 

In an alternative embodiment, such an enhanced pointing device could also 
be a capacitive touchpad, capable of sensing an activating object, such as a finger 
pen, and the like, in close proximity to its surface, as well as when touched to the 
surface. 

15 In a further alternative embodiment, the enhanced pointing device could be 

any device capable of providing X and Y position data, and two or more states 
associated with that data, those states being a base state (corresponding to a mouse 
pointer position with no buttons pressed, or a activating object in close proximity 
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to, but not touching a display screen), a selected state (corresponding to a mouse 
pointer position with the left button clicked, or a activating object touched to the 
display screen of a smart compact device), and zero or more further extended 
states (corresponding, for example, to a mouse with the right button clicked, or a 
activating object with a side button held in). 

In a further alternative embodiment, the enhanced pointing device is an 
inductive pen input system (or inductive sensing system) that is capable of 
providing X and Y position data, a base state, a selected state, and zero or more 
extended states, where the extended states are related to the distance of the 
activating object from the sensor. 

In a further alternative embodiment, the enhanced pointing device is a 
device capable of providing X and Y position data, a base state, a selected state, 
and extended states, where the extended states are selected by a user-controlled 
parameter of the enhanced pointing device. 

In an illustrative embodiment, the pen control system 20 is driven by 
software running on the microprocessor of the smart compact device, the pen 
driver 22. This pen driver 22 converts the pen data 34 from the sensor system into 
pen position messages 36, formatted according to the requirements of the 
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operating system 24. These messages contain both the X and Y position data, and 
flags to signal which one of the states the pen system is reporting. 

In an alternative embodiment, the software running on the smart compact 
device may be structured in a manner other than that shown in FIG. 3, but one that 
reports of the activating object position to the user interface. 

In an illustrative embodiment, the operating system 24 processes the pen 
position message and alters the visible state of any appropriate screen object, 
according to the state of the activating object as reported in the pen position 
message. If the activating object has been held above a screen object for more than 
a short length of time (typically 1 second or less), the operating system will 
change the state of the screen object so that the pop-up data is displayed. FIG. 4 
shows a typical screen object 50, a hyperlink 54. FIG. 5 shows the same screen 
object 50, and its associated pop-up data 52 that has been triggered by holding the 
activating object above the hyperlink 54. 

In an alternative embodiment, the pop-up data can be triggered immediately 
when the activating object is held over the screen object. Such an embodiment can 
be used, for example, to enable a user to easily dial the correct number from the 
contacts directory of a smartphone. 
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FIG. 6 shows a typical small display screen on a smartphone. The display 
screen 60 is showing device status information 62, 64 and a window 66 with 
directory information. Within the window 66, the directory information is shown 
as lines 68, each containing screen objects, such as the contact name 70, the 

5 contact company 72 and an icon 74 intended to trigger the dialing process. An 
alternative to this screen is shown in FIG. 7, where the contact names 70 do not 
have a dial icon associated with them. 

The data displayed on this screen is a summary or abstraction of the data 
held in the smart compact device about each contact. Typically, each contact will 

10 have at least the data illustrated in FIG. 8 stored. The window 66 in FIG. 8 is 
displaying both detailed contact information 80, and control objects in the form of 
icons, which allow dialing the contact 86 and 88, sending email 84 or returning the 
display to the directory listing 82 of FIG. 7. 

In the prior art smart compact devices that use a resistive touch screen as a 

15 pointing device, the process of dialing a contact from the directory would typically 
involve the steps shown in FIG. 9. The user would select the display of the 
directory, tap the desired contact name 70, and then tap the dial icon 88 in the 
detailed display screen. This involves three discrete user actions. 
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In an alternative prior-art embodiment, tapping a dial icon in the directory 
listing (reference numeral 74 in FIG. 6) would dial a number directly. This has two 
key disadvantages. Firstly, there will usually be more than one number or contact 
method associated with each contact, so accessing a detailed screen would still be 
5 necessary. Secondly, the small amount of abstracted data visible on the directory 
screen is not always sufficient to uniquely identify the desired contact. 
0 FIG. 10 illustrates a typical prior-art process for dialing a number when the 

g contact name may be ambiguous. If the desired contact 70 has the same first name 

2 and company as another contact 71, the wrong contact may be selected initially. 

!U 10 Then the user has to click the return icon 82 to return to the directory listing (or 
^ perform a similar 'undo' action). Then the correct desired contact 70 must be 

^ selected, followed by the selection of the dial icon 88 of the correct contact. This 

process takes five discrete user actions. 

In the present invention, the process is simplified for the user. FIG. 1 1 
15 shows a process according to the present invention. The contacts directory is 
selected, the activating object is held stationary above the desired contact name 70, 
which triggers the pop-up data and displays the detailed contact information. As 
part of the same movement the user taps the dial icon 88 to initiate the dialing 
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sequence. This process involves only two discrete user actions, together with a 
natural, intuitive movement of the activating object. 

In the case of ambiguous names, the process of FIG. 12 is followed. Once 
the contacts directory is selected, the activating object is held above the incorrect 
5 contact name 71, triggering the pop-up data with the full contact details. Instead of 
having to tap the return icon 82, the user simply moves the activating object out of 
the proximity range, and then holds it over the correct contact name 70. The pop- 
up data shows the detailed information, and the user can tap on the dial icon 88, 
initiating dialing. The complete process takes only three discrete user actions, 
10 together with a natural intuitive movement of the activating object. 

In an alternative embodiment, the removal of the pop-up data may be 
signalled by another user action, such as the movement of the activating object 
outside the area of the display screen occupied by the pop-up data. 

It will be recognized by those skilled in the art that the present invention is 
15 not limited to use with a contacts directory, but can be used for any application 
that requires an abstracted display of detailed data, and ready access to the detailed 
data. 
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In an alternative embodiment, the pop-up data does not have to take up the 
whole display screen of the smart compact device, but can be displayed on a part 
of the display. FIG. 13 illustrates such a display. The activating object is held 
above the desired contact name 70, which is then highlighted, triggering the pop- 
5 up data containing contact details 90 in the lower section of the screen. Moving 
the activating object to another contact name will cause the pop-up data to change 
to details of the new contact, dialing the contact can be initiated by tapping the dial 
icon 88. This embodiment has the advantage that the context of the contact name 
can be seen together with the contact details. In this embodiment the pop-up data 
10 is not removed when the activating object is moved away from the display screen, 
but remains until an action icon has been selected, pop-up data has been triggered 
from another contact name, or another application is selected on the smart 
compact device. 

In a further alternative embodiment, the directory listing takes the form of 
15 FIG. 14. Here, the display screen in the directory application always has a listing 
92, and pop-up data 90 associated with the highlighted contact 70. Holding and 
moving the activating object in proximity to the display above the listing causes 
the highlight to move, changing the pop-up data to match the highlighted contact. 

15 
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This allows the user to ensure that the right contact is selected from several 
ambiguous contacts. Clicking the activating object on the highlighted contact 
brings up the full information screen of FIG. 8, allowing the user to select the 
required contact method. This embodiment addresses both key disadvantages of 
5 the prior art, while being easy and intuitive for the user. 

It is a further advantage of this embodiment that movement of the 
activating object to the upper or lower extremes of the directory list (Reference 
numeral 92 of FIG. 14) can cause the list to scroll automatically. 

It will be realized by those skilled in the art, that the present invention 
10 applies to all classes of screen objects, whether they are used in the software 
application user interface, as part of a the operating system user interface, or in a 
smart compact device where the software is not structured in this way, but 
contains a user interface and an enhanced pointing device. It will also be 
appreciated that the novel use of the combination of an enhanced pointing device 
15 and pop-up data greatly enhances the usability of the user interface on a smart 
compact device, in a way that has not previously been demonstrated. 

Alternatively, the behavior of the screen object may be modified by the 
state information returned by the enhanced pointing device with the X and Y 
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position data, so that more than one control action maybe produced from one 
control object. 

Alternatively, the pop-up data for a screen object may be triggered by a 
further extended state of the enhanced pointing device, distinct from the base and 
5 selected states. 

In an illustrative embodiment, the exact method of triggering the pop-up 
data, the duration of any timings associated with triggering the pop-up data, the 
range from the display screen that is defined as proximity and the form of any 
pop-up data will be determined by usability testing so as to maximise the benefit 
10 to the user in a given application. 

While embodiments and applications of this invention have been shown 
and described, it would be apparent to those skilled in the art that many more 
modifications than mentioned above are possible without departing from the 
inventive concepts herein. The invention, therefore, is not to be restricted except 
15 in the spirit of the appended claims. 
What is claimed is: 
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